Binaural horizontal perspective hands-on simulator

ABSTRACT

The present invention hands-on simulator system discloses a three dimension display system comprising a three dimensional horizontal perspective display and a 3-D audio system such as binaural simulation to lend realism to the three dimensional display. The three dimensional display system can futher comprise a second display, together with a curvilinear blending display section to merge the various images. The multi-plane display surface can accommodate the viewer by adjusting the various images and 3-D sound according to the viewer&#39;s eyepoint and earpoint locations. The present invention hands-on simulator system can project horizontal perspective images into the open space and a peripheral device that allow the end user to manipulate the images with hands or hand-held tools.

This application claims priority from U.S. provisional applications Ser.No. 60/576, 187 filed Jun. 1, 2004, entitled “Multi plane horizontalperspective display”; Ser. No. 60/576,189 filed Jun. 1, 2004, entitled“Multi plane horizontal perspective hand on simulator”; Ser. No. 60/576,182 filed Jun. 1, 2004, entitled “Binaural horizontal perspectivedisplay”; and Ser. No. 60/576,181 filed Jun. 1, 2004, entitled “Binauralhorizontal perspective hand on simulator” which are incorporated hereinby reference. This application is related to co-pending application Ser.No. 11/098,681 filed Apr. 4, 2005, entitled “Horizontal projectiondisplay”; Ser. No. 11/098,685 filed Apr. 4, 2005, entitled “Horizontalprojection display”, Ser. No. 11/098,667 filed Apr. 4, 2005, entitled“Horizontal projection hands-on simulator”; Ser. No. 11/098,682 filedApr. 4, 2005, entitled “Horizontal projection hands-on simulator”;“Multi plane horizontal perspective display” filed May 27, 2005; “Multiplane horizontal perspective hand on simulator” filed May 27, 2005;“Binaural horizontal perspective display” filed May 27, 2005; and“Binaural horizontal perspective hand on simulator” filed May 27, 2005.

FIELD OF INVENTION

This invention relates to a three-dimensional simulator system, and inparticular, to a multi-plane hands-on computer simulator system capableof operator's interaction.

BACKGROUND OF THE INVENTION

Three dimensional (3D) capable electronics and computing hardwaredevices and real-time computer-generated 3D computer graphics have beena popular area of computer science for the past few decades, withinnovations in visual, audio and tactile systems. Much of the researchin this area has produced hardware and software products that arespecifically designed to generate greater realism and more naturalcomputer-human interfaces. These innovations have significantly enhancedand simplified the end-user's computing experience.

Ever since humans began to communicate through pictures, they faced adilemma of how to accurately represent the three-dimensional world theylived in. Sculpture was used to successfully depict three-dimensionalobjects, but was not adequate to communicate spatial relationshipsbetween objects and within environments. To do this, early humansattempted to “flatten” what they saw around them onto two-dimensional,vertical planes (e.g. paintings, drawings, tapestries, etc.). Sceneswhere a person stood upright, surrounded by trees, were renderedrelatively successfully on a vertical plane. But how could theyrepresent a landscape, where the ground extended out horizontally fromwhere the artist was standing, as far as the eye could see?

The answer is three dimensional illusions. The two dimensional picturesmust provide a numbers of cues of the third dimension to the brain tocreate the illusion of three dimensional images. This effect of thirddimension cues can be realistically achievable due to the fact that thebrain is quite accustomed to it. The three dimensional real world isalways and already converted into two dimensional (e.g. height andwidth) projected image at the retina, a concave surface at the back ofthe eye. And from this two dimensional image, the brain, throughexperience and perception, generates the depth information to form thethree dimension visual image from two types of depth cues: monocular(one eye perception) and binocular (two eye perception). In general,binocular depth cues are innate and biological while monocular depthcues are learned and environmental.

The major binocular depth cues are convergence and retinal disparity.The brain measures the amount of convergence of the eyes to provide arough estimate of the distance since the angle between the line of sightof each eye is larger when an object is closer. The disparity of theretinal images due to the separation of the two eyes is used to createthe perception of depth. The effect is called stereoscopy where each eyereceives a slightly different view of a scene, and the brain fuses themtogether using these differences to determine the ratio of distancesbetween nearby objects.

Binocular cues are very powerful perception of depth. However, there arealso depth cues with only one eye, called monocular depth cues, tocreate an impression of depth on a flat image. The major monocular cuesare: overlapping, relative size, linear perspective and light andshadow. When an object is viewed partially covered, this pattern ofblocking is used as a cue to determine that the object is farther away.When two objects known to be the same size and one appears smaller thanthe other, this pattern of relative size is used as a cue to assume thatthe smaller object is farther away. The cue of relative size alsoprovides the basis for the cue of linear perspective where the fartheraway the lines are from the observer, the closer together they willappear since parallel lines in a perspective image appear to convergetowards a single point. The light falling on an object from a certainangle could provide the cue for the form and depth of an object. Thedistribution of light and shadow on objects is a powerful monocular cuefor depth provided by the biologically correct assumption that lightcomes from above.

Perspective drawing, together with relative size, is most often used toachieve the illusion of three dimension depth and spatial relationshipson a flat (two dimension) surface, such as paper or canvas. Throughperspective, three dimension objects are depicted on a two dimensionplane, but “trick” the eye into appearing to be in three dimensionspace. The first theoretical treatise for constructing perspective,Depictura, was published in the early 1400's by the architect, LeoneBattista Alberti. Since the introduction of his book, the details behind“general” perspective have been very well documented. However, the factthat there are a number of other types of perspectives is not wellknown. Some examples are military, cavalier, isometric, and dimetric, asshown at the top of FIG. 1.

Of special interest is the most common type of perspective, calledcentral perspective, shown at the bottom left of FIG. 1. Centralperspective, also called one-point perspective, is the simplest kind of“genuine” perspective construction, and is often taught in art anddrafting classes for beginners. FIG. 2 further illustrates centralperspective. Using central perspective, the chess board and chess pieceslook like three dimension objects, even though they are drawn on a twodimensional flat piece of paper. Central perspective has a centralvanishing point, and rectangular objects are placed so their front sidesare parallel to the picture plane. The depth of the objects isperpendicular to the picture plane. All parallel receding edges runtowards a central vanishing point. The viewer looks towards thisvanishing point with a straight view. When an architect or artistcreates a drawing using central perspective, they must use a single-eyeview. That is, the artist creating the drawing captures the image bylooking through only one eye, which is perpendicular to the drawingsurface.

The vast majority of images, including central perspective images, aredisplayed, viewed and captured in a plane perpendicular to the line ofvision. Viewing the images at angle different from 90° would result inimage distortion, meaning a square would be seen as a rectangle when theviewing surface is not perpendicular to the line of vision.

Central perspective is employed extensively in 3D computer graphics, fora myriad of applications, such as scientific, data visualization,computer-generated prototyping, special effects for movies, medicalimaging, and architecture, to name just a few. One of the most commonand well-known 3D computing applications is 3D gaming, which is usedhere as an example, because the core concepts used in 3D gaming extendto all other 3D computing applications.

FIG. 3 is a simple illustration, intended to set the stage by listingthe basic components necessary to achieve a high level of realism in 3Dsoftware applications. At its highest level, 3D game developmentconsists of four essential components:

-   -   1. Design: Creation of the game's story line and game play    -   2. Content: The objects (figures, landscapes, etc.) that come to        life during game play    -   3. Artificial Intelligence (AI): Controls interaction with the        content during game play    -   4. Real-time computer-generated 3D graphics engine (3D graphics        engine):

Manages the design, content, and AI data. Decides what to draw, and howto draw it, then renders (displays) it on a computer monitor A personusing a 3D application, such as a game, is in fact running software inthe form of a real-time computer-generated 3D graphics engine. One ofthe engine's key components is the renderer. Its job is to take 3Dobjects that exist within computer-generated world coordinates x, y, z,and render (draw/display) them onto the computer monitor's viewingsurface, which is a flat (2D) plane, with real world coordinates x, y.

FIG. 4 is a representation of what is happening inside the computer whenrunning a 3D graphics engine. Within every 3D game there exists acomputer-generated 3D “world.” This world contains everything that couldbe experienced during game play. It also uses the Cartesian coordinatesystem, meaning it has three spatial dimensions x, y, and z. These threedimensions are referred to as “virtual world coordinates”. Game play fora typical 3D game might begin with a computer-generated-3D earth and acomputer-generated-3D satellite orbiting it. The virtual worldcoordinate system enables the earth and satellite to be properlypositioned in computer-generated x, y, z space.

As they move through time, the satellite and earth must stay properlysynchronized. To accomplish this, the 3D graphics engine creates afourth universal dimension for computer-generated time, t. For everytick of time t, the 3D graphics engine regenerates the satellite at itsnew location and orientation as it orbits the spinning earth. Therefore,a key job for a 3D graphics engine is to continuously synchronize andregenerate all 3D objects within all four computer-generated dimensionsx, y, z, and t.

FIG. 5 is a conceptual illustration of what happens inside the computerwhen an end-user is playing, i.e. running, a first-person 3Dapplication. First-person means that the computer monitor is much like awindow, through which the person playing the game views thecomputer-generated world. To generate this view, the 3D graphics enginerenders the scene from the point of view of the eye of acomputer-generated person. The computer-generated person can be thoughtof as a computer-generated or “virtual” simulation of the “real” personactually playing the game.

While running a 3D application the real person, i.e. the end-user, viewsonly a small segment of the entire 3D world at any given time. This isdone because it is computationally expensive for the computer's hardwareto generate the enormous number of 3D objects in a typical 3Dapplication, the majority of which the end-user is not currently focusedon. Therefore, a critical job for the 3D graphics engine is to minimizethe computer hardware's computational burden by drawing/rendering aslittle information as absolutely necessary during each tick ofcomputer-generated time t.

The boxed-in area in FIG. 5 conceptually represents how a 3D graphicsengine minimizes the hardware's burden. It focuses computationalresources on extremely small areas of information as compared to the 3Dapplications entire world. In this example, it is a “computer-generated”polar bear cub being observed by a “computer-generated” virtual person.Because the end user is running in first-person everything thecomputer-generated person sees is rendered onto the end-user's monitor,i.e. the end user is looking through the eye of the computer-generatedperson.

In FIG. 5 the computer-generated person is looking through only one eye;in other words, an one-eyed view. This is because the 3D graphicsengine's renderer uses central perspective to draw/render 3D objectsonto a 2D surface, which requires viewing through only one eye. The areathat the computer-generated person sees with a one-eye view is calledthe “view volume”, and the computer-generated 3D objects within thisview volume are what actually get rendered to the computer monitor's 2Dviewing surface.

FIG. 6 illustrates a view volume in more detail. A view volume is asubset of a “camera model”. A camera model is a blueprint that definesthe characteristics of both the hardware and software of a 3D graphicsengine. Like a very complex and sophisticated automobile engine, a 3Dgraphics engine consist of so many parts that their camera models areoften simplified to illustrate only the essential elements beingreferenced.

The camera model depicted in FIG. 6 shows a 3D graphics engine usingcentral perspective to render computer-generated 3D objects to acomputer monitor's vertical, 2D viewing surface. The view volume shownin FIG. 6, although more detailed, is the same view volume representedin FIG. 5. The only difference is semantics because a 3D graphics enginecalls the computer-generated person's one-eye view a camera point (hencecamera model).

Every component of a camera model is called an “element”. In oursimplified camera model, the element called near clip plane is the 2Dplane onto which the x, y, z coordinates of the 3D objects within theview volume will be rendered. Each projection line starts at the camerapoint, and ends at a x, y. z coordinate point of a virtual 3D objectwithin the view volume. The 3D graphics engine then determines where theprojection line intersects the near clip-plane and the x and y pointwhere this intersection occurs is rendered onto the near clip-plane.Once the 3D graphics engine's renderer completes all necessarymathematical projections, the near clip plane is displayed on the 2Dviewing surface of the computer monitor, as shown in FIG. 6.

The basic of prior art 3D computer graphics is the central perspectiveprojection. 3D central perspective projection, though offering realistic3D illusion, has some limitations is allowing the user to have hands-oninteraction with the 3D display.

There is a little known class of images that we called it “horizontalperspective” where the image appears distorted when viewing head on, butdisplaying a three dimensional illusion when viewing from the correctviewing position. In horizontal perspective, the angle between theviewing surface and the line of vision is preferably 45° but can bealmost any angle, and the viewing surface is preferably horizontal(wherein the name “horizontal perspective”), but it can be any surface,as long as the line of vision forming a not-perpendicular angle to it.

Horizontal perspective images offer realistic three dimensionalillusion, but are little known primarily due to the narrow viewinglocation (the viewer's eyepoint has to be coincide precisely with theimage projection eyepoint), and the complexity involving in projectingthe two dimensional image or the three dimension model into thehorizontal perspective image.

The generation of horizontal perspective images requires considerablymore expertise to create than conventional perpendicular images. Theconventional perpendicular images can be produced directly from theviewer or camera point. One need simply open one's eyes or point thecamera in any direction to obtain the images. Further, with muchexperience in viewing three dimensional depth cues from perpendicularimages, viewers can tolerate significant amount of distortion generatedby the deviations from the camera point. In contrast, the creation of ahorizontal perspective image does require much manipulation.Conventional camera, by projecting the image into the planeperpendicular to the line of sight, would not produce a horizontalperspective image. Making a horizontal drawing requires much effort andvery time consuming. Further, since human has limited experience withhorizontal perspective images, the viewer's eye must be positionedprecisely where the projection eyepoint point is to avoid imagedistortion. And therefore horizontal perspective, with its difficulties,has received little attention.

For realistic three dimensional simulation, binaural or threedimensional audio simulation is also needed.

SUMMARY OF THE INVENTION

The present invention recognizes that the personal computer is perfectlysuitable for horizontal perspective display. It is personal, thus it isdesigned for the operation of one person, and the computer, with itspowerful microprocessor, is well capable of rendering various horizontalperspective images to the viewer. Further, horizontal perspective offersopen space display of 3D images, thus allowing the hands-on interactionof the end users.

Thus the present invention discloses a multi-plane hands-on simulatorsystem comprising at least two display surfaces, one of which displayinga three dimensional horizontal perspective images. The other displaysurfaces can display two dimensional images, or preferably threedimensional central perpective images. Further, the display surfaces canhave a curvilinear blending display section to merge the various images.The multi-plane hands-on simulator can comprise various cameraeyepoints, one for the horizontal perspective images, one for thecentral perspective images, and optionally one for the curvilinearblending display surface. The multi-plane display surface can furtheradjust the various images to accommodate the position of the viewer. Bychanging the displayed images to keep the camera eyepoints of thehorizontal perspective and central perspective images in the sameposition as the viewer's eye point, the viewer's eye is alwayspositioned at the proper viewing position to perceive the threedimensional illusion, thus minimizing viewer's discomfort anddistortion. The display can accept manual input such as a computermouse, trackball, joystick, tablet, etc. to re-position the horizontalperspective images. The display can also automatically re-position theimages based on an input device automatically providing the viewer'sviewpoint location. The multi-plane hands-on simulator system canproject horizontal perspective images into the open space and aperipheral device that allow the end user to manipulate the images withhands or hand-held tools. Further, the display is also included threedimensional audio such as binaural simulation to lend realism to thethree dimensional display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the various perspective drawings.

FIG. 2 shows a typical central perspective drawing.

FIG. 3 shows 3D software application.

FIG. 4 shows 3D application running on PC.

FIG. 5 shows 3D application in first person.

FIG. 6 shows central perspective camera model.

FIG. 7 shows the comparison of central perspective (Image A) andhorizontal perspective (Image B).

FIG. 8 shows the central perspective drawing of three stacking blocks.

FIG. 9 shows the horizontal perspective drawing of three stackingblocks.

FIG. 10 shows the method of drawing a horizontal perspective drawing.

FIG. 11 shows a horizontal perspective display and a viewer inputdevice.

FIG. 12 shows a horizontal perspective display, a computational deviceand a viewer input device.

FIG. 13 shows a computer monitor.

FIG. 14 shows a monitor's phosphor layer indicating of an incorrectlocation of image.

FIG. 15 shows a monitor's viewing surface indicating of a correctlocation of image.

FIG. 16 shows a reference plane x, y, z coordinates.

FIG. 17 shows the location of an angled camera point.

FIG. 18 shows the mapping of the horizontal plane to a reference plane.

FIG. 19 shows the comfort plane.

FIG. 20 shows the hands-on volume.

FIG. 21 shows the inner plane.

FIG. 22 shows the bottom plane.

FIG. 23 shows the inner access volume.

FIG. 24 shows the angled camera mapped to the end-user's eye

FIG. 25 shows mapping of the 3-d object onto the horizontal plane.

FIG. 26 shows the two-eye view.

FIG. 27 shows the simulation time of the horizontal perspective.

FIG. 28 shows the horizontal plane.

FIG. 29 shows the 3D peripherals.

FIG. 30 shows an open-access camera model.

FIG. 31 shows the concept of object recognition.

FIG. 32 shows the 3D audio combination with object recognition.

FIG. 33 shows another open access camera model.

FIG. 34 shows another open access camera model

FIG. 35 shows the mapping of virtual attachments to end of tools.

FIG. 36 shows the multi-plane and multi-view device.

FIG. 37 shows an open access camera model.

FIG. 38 shows another multi-plane device.

DETAILED DESCRIPTION OF THE INVENTION

The new and unique inventions described in this document build uponprior art by taking the current state of real-time computer-generated 3Dcomputer graphics, 3D sound, and tactile computer-human interfaces to awhole new level of reality and simplicity. More specifically, these newinventions enable real-time computer-generated 3D simulations to coexistin physical space and time with the end-user and with other real-worldphysical objects. This capability dramatically improves upon theend-user's visual, auditory and tactile computing experience byproviding direct physical interactions with 3D computer-generatedobjects and sounds. This unique ability is useful in nearly everyconceivable industry including, but not limited to, electronics,computers, biometrics, medical, education, games, movies, science,legal, financial, communication, law enforcement, national security,military, print media, television, advertising, trade show, datavisualization, computer-generated reality, animation, CAD/CAE/CAM,productivity software, operating systems, and more.

The present invention discloses a multi-plane horizontal perspectivehands-on simulator comprising at least two display surfaces, one ofwhich capable of projecting three dimensional illusion based onhorizontal perspective projection.

In general, the present invention horizontal perspective hands-onsimulator can be used to display and interact with three dimensionalimages and has obvious utility to many industrial applications such asmanufacturing design reviews, ergonomic simulation, safety and training,video games, cinematography, scientific 3D viewing, and medical andother data displays.

Horizontal perspective is a little-known perspective, of which we foundonly two books that describe its mechanics: Stereoscopic Drawing (®1990)and How to Make Anaglyphs (®1979, out of print). Although these booksdescribe this obscure perspective, they do not agree on its name. Thefirst book refers to it as a “free-standing anaglyph,” and the second, a“phantogram.” Another publication called it “projective anaglyph” (U.S.Pat. No. 5,795,154 by G. M. Woods, Aug. 18, 1998). Since there is noagreed-upon name, we have taken the liberty of calling it “horizontalperspective.” Normally, as in central perspective, the plane of vision,at right angle to the line of sight, is also the projected plane of thepicture, and depth cues are used to give the illusion of depth to thisflat image. In horizontal perspective, the plane of vision remains thesame, but the projected image is not on this plane. It is on a planeangled to the plane of vision. Typically, the image would be on theground level surface. This means the image will be physically in thethird dimension relative to the plane of vision. Thus horizontalperspective can be called horizontal projection.

In horizontal perspective, the object is to separate the image from thepaper, and fuse the image to the three dimension object that projectsthe horizontal perspective image. Thus the horizontal perspective imagemust be distorted so that the visual image fuses to form the freestanding three dimensional figure. It is also essential the image isviewed from the correct eye points, otherwise the three dimensionalillusion is lost. In contrast to central perspective images which haveheight and width, and project an illusion of depth, and therefore theobjects are usually abruptly projected and the images appear to be inlayers, the horizontal perspective images have actual depth and width,and illusion gives them height, and therefore there is usually agraduated shifting so the images appear to be continuous.

FIG. 7 compares key characteristics that differentiate centralperspective and horizontal perspective. Image A shows key pertinentcharacteristics of central perspective, and Image B shows key pertinentcharacteristics of horizontal perspective.

In other words, in Image A, the real-life three dimension object (threeblocks stacked slightly above each other) was drawn by the artistclosing one eye, and viewing along a line of sight perpendicular to thevertical drawing plane. The resulting image, when viewed vertically,straight on, and through one eye, looks the same as the original image.

In Image B, the real-life three dimension object was drawn by the artistclosing one eye, and viewing along a line of sight 45° to the horizontaldrawing plane. The resulting image, when viewed horizontally, at 45° andthrough one eye, looks the same as the original image.

One major difference between central perspective showing in Image A andhorizontal perspective showing in Image B is the location of the displayplane with respect to the projected three dimensional image. Inhorizontal perspective of Image B, the display plane can be adjusted upand down, and therefore the projected image can be displayed in the openair above the display plane, i.e. a physical hand can touch (or morelikely pass through) the illusion, or it can be displayed under thedisplay plane, i.e. one cannot touch the illusion because the displayplane physically blocks the hand. This is the nature of horizontalperspective, and as long as the camera eyepoint and the viewer eyepointis at the same place, the illusion is present. In contrast, in centralperspective of Image A, the three dimensional illusion is likely to beonly inside the display plane, meaning one cannot touch it. To bring thethree dimensional illusion outside of the display plane to allow viewerto touch it, the central perspective would need elaborate display schemesuch as surround image projection and large volume.

FIGS. 8 and 9 illustrate the visual difference between using central andhorizontal perspective. To experience this visual difference, first lookat FIG. 8, drawn with central perspective, through one open eye. Holdthe piece of paper vertically in front of you, as you would atraditional drawing, perpendicular to your eye. You can see that centralperspective provides a good representation of three dimension objects ona two dimension surface.

Now look at FIG. 9, drawn using horizontal perspective, by sifting atyour desk and placing the paper lying flat (horizontally) on the desk infront of you. Again, view the image through only one eye. This puts yourone open eye, called the eye point at approximately a 45° angle to thepaper, which is the angle that the artist used to make the drawing. Toget your open eye and its line-of-sight to coincide with the artist's,move your eye downward and forward closer to the drawing, about sixinches out and down and at a 45° angle. This will result in the idealviewing experience where the top and middle blocks will appear above thepaper in open space.

Again, the reason your one open eye needs to be at this precise locationis because both central and horizontal perspective not only defines theangle of the line of sight from the eye point; they also define thedistance from the eye point to the drawing. This means that FIGS. 8 and9 are drawn with an ideal location and direction for your open eyerelative to the drawing surfaces. However, unlike central perspectivewhere deviations from position and direction of the eye point createlittle distortion, when viewing a horizontal perspective drawing, theuse of only one eye and the position and direction of that eye relativeto the viewing surface are essential to seeing the open space threedimension horizontal perspective illusion.

FIG. 10 is an architectural-style illustration that demonstrates amethod for making simple geometric drawings on paper or canvas utilizinghorizontal perspective. FIG. 10 is a side view of the same three blocksused in FIGS. 9. It illustrates the actual mechanics of horizontalperspective. Each point that makes up the object is drawn by projectingthe point onto the horizontal drawing plane. To illustrate this, FIG. 10shows a few of the coordinates of the blocks being drawn on thehorizontal drawing plane through projection lines. These projectionlines start at the eye point (not shown in FIG. 10 due to scale),intersect a point on the object, then continue in a straight line towhere they intersect the horizontal drawing plane, which is where theyare physically drawn as a single dot on the paper When an architectrepeats this process for each and every point on the blocks, as seenfrom the drawing surface to the eye point along the line-of-sight thehorizontal perspective drawing is complete, and looks like FIG. 9.

Notice that in FIG. 10, one of the three blocks appears below thehorizontal drawing plane. With horizontal perspective, points locatedbelow the drawing surface are also drawn onto the horizontal drawingplane, as seen from the eye point along the line-of-site. Therefore whenthe final drawing is viewed, objects not only appear above thehorizontal drawing plane, but may also appear below it as well—givingthe appearance that they are receding into the paper. If you look againat FIG. 9, you will notice that the bottom box appears to be below, orgo into, the paper, while the other two boxes appear above the paper inopen space.

The generation of horizontal perspective images requires considerablymore expertise to create than central perspective images. Even thoughboth methods seek to provide the viewer the three dimension illusionthat resulted from the two dimensional image, central perspective imagesproduce directly the three dimensional landscape from the viewer orcamera point. In contrast, the horizontal perspective image appearsdistorted when viewing head on, but this distortion has to be preciselyrendered so that when viewing at a precise location, the horizontalperspective produces a three dimensional illusion.

The horizontal perspective display system promotes horizontalperspective projection viewing by providing the viewer with the means toadjust the displayed images to maximize the illusion viewing experience.By employing the computation power of the microprocessor and a real timedisplay, the horizontal perspective display is shown in FIG. 11,comprising a real time electronic display 100 capable of re-drawing theprojected image, together with a viewer's input device 102 to adjust thehorizontal perspective image. By re-display the horizontal perspectiveimage so that its projection eyepoint coincides with the eyepoint of theviewer, the horizontal perspective display can ensure the minimumdistortion in rendering the three dimension illusion from the horizontalperspective method. The input device can be manually operated where theviewer manually inputs his or her eyepoint location, or change theprojection image eyepoint to obtain the optimum three dimensionalillusion. The input device can also be automatically operated where thedisplay automatically tracks the viewer's eyepoint and adjust theprojection image accordingly. The horizontal perspective display removesthe constraint that the viewers keeping their heads in relatively fixedpositions, a constraint that create much difficulty in the acceptance ofprecise eyepoint location such as horizontal perspective or hologramdisplay.

The horizontal perspective display system, shown in FIG. 12, can furthercomprise a computation device 110 in addition to the real timeelectronic display device 100 and projection image input device 112providing input to the computational device 110 to calculating theprojectional images for display to providing a realistic, minimumdistortion three dimensional illusion to the viewer by coincide theviewer's eyepoint with the projection image eyepoint. The system canfurther comprise an image enlargement/reduction input device 115, or animage rotation input device 117, or an image movement device 119 toallow the viewer to adjust the view of the projection images.

The horizontal perspective display system promotes horizontalperspective projection viewing by providing the viewer with the means toadjust the displayed images to maximize the illusion viewing experience.By employing the computation power of the microprocessor and a real timedisplay, the horizontal perspective display, comprising a real timeelectronic display capable of re-drawing the projected image, togetherwith a viewer's input device to adjust the horizontal perspective image.By re-display the horizontal perspective image so that its projectioneyepoint coincides with the eyepoint of the viewer, the horizontalperspective display of the present invention can ensure the minimumdistortion in rendering the three dimension illusion from the horizontalperspective method. The input device can be manually operated where theviewer manually inputs his or her eyepoint location, or change theprojection image eyepoint to obtain the optimum three dimensionalillusions. The input device can also be automatically operated where thedisplay automatically tracks the viewer's eyepoint and adjust theprojection image accordingly. The horizontal perspective display systemremoves the constraint that the viewers keeping their heads inrelatively fixed positions, a constraint that create much difficulty inthe acceptance of precise eyepoint location such as horizontalperspective or hologram display.

The horizontal perspective display system can further a computationdevice in addition to the real time electronic display device andprojection image input device providing input to the computationaldevice to calculating the projectional images for display to providing arealistic, minimum distortion three dimensional illusion to the viewerby coincide the viewer's eyepoint with the projection image eyepoint.The system can further comprise an image enlargement/reduction inputdevice, or an image rotation input device, or an image movement deviceto allow the viewer to adjust the view of the projection images.

The input device can be operated manually or automatically. The inputdevice can detect the position and orientation of the viewer eyepoint,to compute and to project the image onto the display according to thedetection result. Alternatively, the input device can be made to detectthe position and orientation of the viewer's head along with theorientation of the eyeballs. The input device can comprise an infrareddetection system to detect the position the viewer's head to allow theviewer freedom of head movement. Other embodiments of the input devicecan be the triangulation method of detecting the viewer eyepointlocation, such as a CCD camera providing position data suitable for thehead tracking objectives of the invention. The input device can bemanually operated by the viewer, such as a keyboard, mouse, trackball,joystick, or the like, to indicate the correct display of the horizontalperspective display images.

The head or eye-tracking system can comprise a base unit and ahead-mounted sensor on the head of the viewer. The head-mounted sensorproduces signals showing the position and orientation of the viewer inresponse to the viewer's head movement and eye orientation. Thesesignals can be received by the base unit and are used to compute theproper three dimensional projection images. The head or eye trackingsystem can be infrared cameras to capture images of the viewer's eyes.Using the captured images and other techniques of image processing, theposition and orientation of the viewer's eyes can be determined, andthen provided to the base unit. The head and eye tracking can be done inreal time for small enough time interval to provide continous viewer'shead and eye tracking.

The invention described in this document, employing the open spacecharacteristics of the horizontal perspective, together with a number ofnew computer hardware and software elements and processes that togetherto create a “Hands-On Simulator”. In the simplest terms, the Hands-OnSimulator generates a totally new and unique computing experience inthat it enables an end user to interact physically and directly(Hands-On) with real-time computer-generated 3D graphics (Simulations),which appear in open space above the viewing surface of a displaydevice, i.e. in the end user's own physical space.

For the end user to experience these unique hands-on simulations thecomputer hardware viewing surface is situated horizontally, such thatthe end-user's line of sight is at a 45° angle to the surface.Typically, this means that the end user is standing or seatedvertically, and the viewing surface is horizontal to the ground. Notethat although the end user can experience hands-on simulations atviewing angles other than 45° (e.g. 55°, 30° etc.), it is the optimalangle for the brain to recognize the maximum amount of spatialinformation in an open space image. Therefore, for simplicity's sake, weuse “45°” throughout this document to mean “an approximate 45 degreeangle”. Further, while horizontal viewing surface is preferred since itsimulates viewers' experience with the horizontal ground, any viewingsurface could offer similar three dimensional illusion experience. Thehorizontal perspective illusion can appear to be hanging from a ceilingby projecting the horizontal perspective images onto a ceiling surface,or appear to be floating from a wall by projecting the horizontalperspective images onto a vertical wall surface.

The hands-on simulations are generated within a 3D graphics engines'view volume, creating two new elements, the “Hands-On Volume” and the“Inner-Access Volume.”

The Hands-On Volume is situated on and above the physical viewingsurface. Thus the end user can directly, physically manipulatesimulations because they co-inhabit the end-user's own physical space.This 1:1 correspondence allows accurate and tangible physicalinteraction by touching and manipulating simulations with hands orhand-held tools. The Inner-Access Volume is located underneath theviewing surface and simulations within this volume appear inside thephysically viewing device. Thus simulations generated within theInner-Access Volume do not share the same physical space with the enduser and the images therefore cannot be directly, physically manipulatedby hands or hand-held tools. That is, they are manipulated indirectlyvia a computer mouse or a joystick.

This disclosed Hands-On Simulator can lead to the end user's ability todirectly, physically manipulate simulations because they co-inhabit theend-user's own physical space. To accomplish this requires a newcomputing concept where computer-generated world elements have a 1:1correspondence with their physical real-world equivalents; that is, aphysical element and an equivalent computer-generated element occupy thesame space and time. This is achieved by identifying and establishing acommon “Reference Plane”, to which the new elements are synchronized.

Synchronization with the Reference Plane forms the basis to create the1:1 correspondence between the “virtual” world of the simulations, andthe “real” physical world. Among other things, the 1:1 correspondenceinsures that images are properly displayed: What is on and above theviewing surface appears on and above the surface, in the Hands-OnVolume; what is underneath the viewing surface appears below, in theInner-Access Volume. Only if this 1:1 correspondence and synchronizationto the Reference Plane are present can the end user physically anddirectly access and interact with simulations via their hands orhand-held tools.

The present invention simulator further includes a real-timecomputer-generated 3D-graphics engine as generally described above, butusing horizontal perspective projection to display the 3D images. Onemajor different between the present invention and prior art graphicsengine is the projection display. Existing 3D-graphics engine usescentral-perspective and therefore a vertical plane to render its viewvolume while in the present invention simulator, a “horizontal” orientedrendering plane vs. a “vertical” oriented rendering plane is required togenerate horizontal perspective open space images. The horizontalperspective images offer much superior open space access than centralperspective images.

One of the invented elements in the present invention hands-on simulatoris the 1:1 correspondence of the computer-generated world elements andtheir physical real-world equivalents. As noted in the introductionabove, this 1:1 correspondence is a new computing concept that isessential for the end user to physically and directly access andinteract with hands-on simulations. This new concept requires thecreation of a common physical Reference Plane, as well as, the formulafor deriving its unique x, y, z spatial coordinates. To determine thelocation and size of the Reference Plane and its specific coordinatesrequires understanding the following.

A computer monitor or viewing device is made of many physical layers,individually and together having thickness or depth. To illustrate this,FIG. 13 contains a conceptual side-view of typical CRT-type viewingdevice. The top layer of the monitor's glass surface is the physical“View Surface”, and the phosphor layer, where images are made, is thephysical “Image Layer”. The View Surface and the Image Layer areseparate physical layers located at different depths or z coordinatesalong the viewing device's z axis. To display an image the CRT'selectron gun excites the phosphors, which in turn emit photons. Thismeans that when you view an image on a CRT, you are looking along its zaxis through its glass surface, like you would a window, and seeing thelight of the image coming from its phosphors behind the glass.

With a viewing device's z axis in mind, let's display an image on thatdevice using horizontal perspective. In FIG. 14 we use the samearchitectural technique for drawing images with horizontal perspectiveas previously illustrated in FIG. 10. By comparing FIG. 14 and FIG. 10you can see that the middle block in FIG. 14 does not correctly appearon the View Surface. In FIG. 10 the bottom of the middle block islocated correctly on the horizontal drawing/viewing plane, i.e. a pieceof paper's View Surface. But in FIG. 14, the phosphor layer, i.e. wherethe image is made, is located behind the CRT's glass surface. Therefore,the bottom of the middle block is incorrectly positioned behind orunderneath the View Surface.

FIG. 15 shows the proper location of the three blocks on a CRT-typeviewing device. That is, the bottom of the middle block is displayedcorrectly on the View Surface and not on the Image Layer. To make thisadjustment the z coordinates of the View Surface and Image Layer areused by the Simulation Engine to correctly render the image. Thus theunique task of correctly rendering an open space image on the ViewSurface vs. the Image Layer is critical in accurately mapping thesimulation images to the real world space.

It is now clear that a viewing device's View Surface is the correctphysical location to present open space images. Therefore, the ViewSurface, i.e. the top of the viewing device's glass surface, is thecommon physical Reference Plane. But only a subset of the View Surfacecan be the Reference Plane because the entire View Surface is largerthan the total image area. FIG. 16 shows an example of a complete imagebeing displayed on a viewing device's View Surface. That is, the blueimage, including the bear cub, shows the entire image area, which issmaller than the viewing device's View Surface.

Many viewing devices enable the end user to adjust the size of the imagearea by adjusting its x and y value. Of course these same viewingdevices do not provide any knowledge of, or access to, the z axisinformation because it is a completely new concept and to date onlyrequired for the display of open space images. But all three, x, y, z,coordinates are essential to determine the location and size of thecommon physical Reference Plane. The formula for this is: The ImageLayer is given a z coordinate of 0. The View Surface is the distancealong the z axis from the Image Layer the Reference Plane's z coordinateis equal to the View Surface, i.e. its distance from the Image Layer.The x and y coordinates, or size of the Reference Plane, can bedetermined by displaying a complete image on the viewing device andmeasuring the length of its x and y axis.

The concept of the common physical Reference Plane is a new inventiveconcept. Therefore, display manufactures may not supply or even know itscoordinates. Thus a “Reference Plane Calibration” procedure might needto be performed to establish the Reference Plane coordinates. Thiscalibration procedure provides the end user with a number oforchestrated images that s/he interacts. The end-user's response tothese images provides feedback to the Simulation Engine such that it canidentify the correct size and location of the Reference Plane. When theend user is satisfied and completes the procedure the coordinates aresaved in the end user's personal profile.

With some viewing devices the distance between the View Surface andImage Layer is quite short. But no matter how small or large thedistance, it is critical that all Reference Plane x, y, and zcoordinates are determined as close as technically possible.

After the mapping of the “computer-generated” horizontal perspectiveprojection display plane (Horizontal Plane) to the “physical” ReferencePlane x, y, z coordinates, the two elements coexist and are coincidentin time and space; that is, the computer-generated Horizontal Plane nowshares the real-world x, y, z coordinates of the physical ReferencePlane, and they exist at the same time.

You can envision this unique mapping of a computer-generated element anda physical element occupying the same space and time by imagining youare sitting in front of a horizontally oriented computer monitor andusing the Hands-On Simulator. By placing your finger on the surface ofthe monitor, you would touch the Reference Plane (a portion of thephysical View Surface) and the Horizontal Plane (computer-generated) atexactly the same time, In other words, when touching the physicalsurface of the monitor, you are also “touching” its computer-generatedequivalent, the Horizontal Plane, which has been created and mapped bythe Simulation Engine to the same location and time.

One element of the present invention horizontal perspective projectionhands-on simulator is a computer-generated “Angled Camera” point, shownin FIG. 17. The camera point is initially located at an arbitrarydistance from the Horizontal Plane and the camera's line-of-site isoriented at a 45° angle looking through the center. The position of theAngled Camera in relation to the end-user's eye is critical togenerating simulations that appear in open space on and above thesurface of the viewing device.

Mathematically, the computer-generated x, y, z coordinates of the AngledCamera point form the vertex of an infinite “pyramid”, whose sides passthrough the x, y, z coordinates of the Reference/Horizontal Plane. FIG.18 illustrates this infinite pyramid, which begins at the Angled Camerapoint and extending through the Far Clip Plane. There are new planeswithin the pyramid that run parallel to the Reference/Horizontal Plane,which, together with the sides of the pyramid define two new viewvolumes. These unique view volumes are called Hands-On and theInner-Access Volume, and are not shown in FIG. 18. The dimensions ofthese volumes and the planes that define them are based on theirlocations within the pyramid.

FIG. 19 illustrates a plane, called Comfort Plane, together with otherdisplay elements. The Comfort Plane is one of six planes that define thenew Hands-On Volume, and of these planes it is closest to the AngledCamera point and parallel to the Reference Plane. The Comfort Plane isappropriately named because its location within the pyramid determinesthe end-user's personal comfort, i.e. how their eyes, head, body, etc.are situated while viewing and interacting with simulations. The enduser can adjust the location of the Comfort Plane based on theirpersonal visual comfort through a “Comfort Plane Adjustment” procedure.This procedure provides the end user with orchestrated simulationswithin the Hands-On Volume, and enables them to adjust the location ofthe Comfort Plane within the pyramid relative to the Reference Plane.When the end user is satisfied and completes the procedure the locationof the Comfort Plane is saved in the end-user's personal profiles.

The present invention simulator further defines a “Hands-On Volume”,shown in FIG. 20. The Hands-On Volume is where you can reach your handin and physically “touch” a simulation. You can envision this byimagining you are sifting in front of a horizontally oriented computermonitor and using the Hands-On Simulator. If you place your hand severalinches above the surface of the monitor, you are putting your handinside both the physical and computer-generated Hands-On Volume at thesame time. The Hands-On Volume exists within the pyramid and are betweenand inclusive of the Comfort Planes and the Reference/Horizontal Planes.

Where the Hands-On Volume exists on and above the Reference/HorizontalPlane, the Inner-Access Volume exists below or inside the physicalviewing device. For this reason, an end user cannot directly interactwith 3D objects located within the Inner-Access Volume via their hand orhand-held tools. But they can interact in the traditional sense with acomputer mouse, joystick, or other similar computer peripheral. An“Inner Plane” is further defined, located immediately below and areparallel to the Reference/Horizontal Plane within the pyramid as shownin FIG. 21. The Inner Plane, along with the Bottom Plane, is two of thesix planes within the pyramid that define the Inner-Access Volume. TheBottom Plane (shown in FIG. 22) is farthest away from the Angled Camerapoint, but it is not to be mistaken for the Far Clip plane. The BottomPlane is also parallel to the Reference/Horizontal Plane and is one ofthe six planes that define the Inner-Access Volume (FIG. 23). You canenvision the Inner-Access Volume by imagining you are sitting in frontof a horizontally oriented computer monitor and using the Hands-OnSimulator. If you pushed your hand through the physical surface andplaced your hand inside the monitor (which of course is not possible),you would be putting your hand inside the Inner-Access Volume.

The end-user's preferred viewing distance to the bottom of the viewingpyramid determines the location of these planes. One way the end usercan adjust the location of the Bottom Planes is through a “Bottom PlaneAdjustment” procedure. This procedure provides the end user withorchestrated simulations within the Inner-Access Volume and enables themto interact and adjust the location of the Bottom Plane relative to thephysical Reference/Horizontal Plane. When the end user completes theprocedure the Bottom Plane's coordinates are saved in the end-user'spersonal profiles.

For the end user to view open space images on their physical viewingdevice it must be positioned properly, which usually means the physicalReference Plane is placed horizontally to the ground. Whatever theviewing device's position relative to the ground, theReference/Horizontal Plane must be at approximately a 45° angle to theend-user's line-of-sight for optimum viewing. One way the end user mightperform this step is to position their CRT computer monitor on the floorin a stand, so that the Reference/Horizontal Plane is horizontal to thefloor. This example use a CRT-type computer monitor, but it could be anytype of viewing device, placed at approximately a 45° angle to theend-user's line-of-sight.

The real-world coordinates of the “End-User's Eye” and thecomputer-generated Angled Camera point must have a 1:1 correspondence inorder for the end user to properly view open space images that appear onand above the Reference/Horizontal Plane (FIG. 24). One way to do thisis for the end user to supply the Simulation Engine with their eye'sreal-world x, yr. z location and line-of-site information relative tothe center of the physical Reference/Horizontal Plane. For example, theend user tells the Simulation Engine that their physical eye will belocated 12 inches up, and 12 inches back, while looking at the center ofthe Reference/Horizontal Plane. The Simulation Engine then maps thecomputer-generated Angled Camera point to the End-User's Eye pointphysical coordinates and line-of-sight.

The present invention horizontal perspective hands-on simulator employsthe horizontal perspective projection to mathematically projected the 3Dobjects to the Hands-On and Inner-Access Volumes. The existence of aphysical Reference Plane and the knowledge of its coordinates areessential to correctly adjusting the Horizontal Plane's coordinatesprior to projection. This adjustment to the Horizontal Plane enablesopen space images to appear to the end user on the View Surface vs. theImage Layer by taking into account the offset between the Image Layerand the View Surface, which are located at different values along theviewing device's z axis.

As a projection line in either the Hands-On and Inner-Access Volumeintersects both an object point and the offset Horizontal Plane, thethree dimensional x, y, z point of the object becomes a two-dimensionalx, y point of the Horizontal Plane (see FIG. 25). Projection lines oftenintersect more than one 3D object coordinate, but only one object x, y,z coordinate along a given projection line can become a Horizontal Planex, y point. The formula to determine which object coordinate becomes apoint on the Horizontal Plane is different for each volume. For theHands-On Volume it is the object coordinate of a given projection linethat is farthest from the Horizontal Plane. For the Inner-Access Volumeit is the object coordinate of a given projection line that is closestto the Horizontal Plane. In case of a tie, i.e. if a 3D object pointfrom each volume occupies the same 2D point of the Horizontal Plane, theHands-On Volume's 3D object point is used.

FIG. 25 is an illustration of the present invention Simulation Enginethat includes the new computer-generated and real physical elements asdescribed above. It also shows that a real-world element and itscomputer-generated equivalent are mapped 1:1 and together share a commonReference Plane. The full implementation of this Simulation Engineresults in a Hands-On Simulator with real-time computer-generated3D-graphics appearing in open space on and above a viewing device'ssurface, which is oriented approximately 45° to the end-user'sline-of-sight.

The Hands-On Simulator further involves adding completely new elementsand processes and existing stereoscopic 3D computer hardware. The resultin a Hands-On Simulator with multiple views or “Multi-View” capability.Multi-View provides the end user with multiple and/or separate left-andright-eye views of the same simulation.

To provide motion, or time-related simulation, the simulator furtherincludes a new computer-generated “time dimension” element, called“SI-time”. SI is an acronym for “Simulation Image” and is one completeimage displayed on the viewing device. SI-Time is the amount of time theSimulation Engine uses to completely generate and display one SimulationImage. This is similar to a movie projector where 24 times a second itdisplays an image. Therefore, 1/24 of a second is required for one imageto be displayed by the projector But SI-Time is variable, meaning thatdepending on the complexity of the view volumes it could take 1/120^(th)or ½ a second for the Simulation Engine to complete just one SI.

The simulator also includes a new computer-generated “time dimension”element, called “EV-time” and is the amount of time used to generate aone “Eye-View”. For example, let's say that the Simulation Engine needsto create one left-eye view and one right-eye view for purposes ofproviding the end user with a stereoscopic 3D experience. If it takesthe Simulation Engine ½ a second to generate the left-eye view then thefirst EV-Time period is ½ a second. If it takes another ½ second togenerate the right-eye view then the second EV-Time period is also ½second. Since the Simulation Engine was generating a separate left andright eye view of the same Simulation Image the total SI-Time is onesecond. That is, the first EV-Time was ½ second and the second EV-Timewas also ½ second making a total SI-Time of one second.

FIG. 26 helps illustrate these two new time dimension elements. It is aconceptual drawing of what is occurring inside the Simulation Enginewhen it is generating a two-eye view of a Simulated Image. Thecomputer-generated person has both eyes open, a requirement forstereoscopic 3D viewing, and therefore sees the bear cub from twoseparate vantage points, i.e. from both a right-eye view and a left-eyeview. These two separate views are slightly different and offset becausethe average person's eyes are about 2 inches apart. Therefore, each eyesees the world from a separate point in space and the brain puts themtogether to make a whole image. This is how and why we see the realworld in stereoscopic 3D.

FIG. 27 is a very high-level Simulation Engine blueprint focusing on howthe computer-generated person's two eye views are projected onto theHorizontal Plane and then displayed on a stereoscopic 3D capable viewingdevice. FIG. 26 represents one complete SI-Time period. If we use theexample from step 3 above, SI-Time takes one second. During this onesecond of SI-Time the Simulation Engine needs to generate two differenteye views, because in this example the stereoscopic 3D viewing devicerequires a separate left- and right-eye view. There are existingstereoscopic 3D viewing devices that require more than a separate left-and right-eye view. But because the method described here can generatemultiple views it works for these devices as well.

The illustration in the upper left of FIG. 27 shows the Angled Camerapoint for the right eye at time-element “EV-Time-1”, which means thefirst Eye-View time period or the first eye-view to be generated. So inFIG. 27, EV-Time-1 is the time period used by the Simulation Engine tocomplete the first eye (right-eye) view of the computer-generatedperson. This is the job for this step, which is within EV-Time-1, andusing the Angled Camera at coordinate x, y, z, the Simulation Enginecompletes the rendering and display of the right-eye view of a givenSimulation Image.

Once the first eye (right-eye) view is complete, the Simulation Enginestarts the process of rendering the computer-generated person's secondeye (left-eye) view. The illustration in the lower left of FIG. 27 showsthe Angled Camera point for the left eye at time element “EV-Time-2”.That is, this second eye view is completed during EV-Time-2. But beforethe rendering process can begin, step 5 makes an adjustment to theAngled Camera point. This is illustrated in FIG. 27 by the left eye's xcoordinate being incremented by two inches. This difference between theright eye's x value and the left eye's x+2″ is what provides thetwo-inch separation between the eyes, which is required for stereoscopic3D viewing.

The distances between people's eyes vary but in the above example we areusing the average of 2 inches. It is also possible for the end user tosupply the Simulation Engine with their personal eye separation value.This would make the x value for the left and right eyes highly accuratefor a given end user and thereby improve the quality of theirstereoscopic 3D view.

Once the Simulation Engine has incremented the Angled Camera point's xcoordinate by two inches, or by the personal eye separation valuesupplied by the end user, it completes the rendering and display of thesecond (left-eye) view. This is done by the Simulation Engine within theEV-Time-2 period using the Angled Camera point coordinate x±2″, y, z andthe exact same Simulation Image rendered. This completes one SI-Timeperiod.

Depending on the stereoscopic 3D viewing device used, the SimulationEngine continues to display the left- and right-eye images, as describedabove, until it needs to move to the next SI-Time period. The job ofthis step is to determine if it is time to move to a new SI-Time period,and if it is, then increment SI-Time. An example of when this may occuris if the bear cub moves his paw or any part of his body Then a new andsecond Simulated Image would be required to show the bear cub in its newposition. This new Simulated Image of the bear cub, in a slightlydifferent location, gets rendered during a new SI-Time period orSI-Time-2. This new SI-time-2 period will have its own EV-Time-1 andEV-Time-2, and therefore the simulation steps described above will berepeated during SI-time-2. This process of generating multiple views viathe nonstop incrementing of SI-Time and its EV-Times continues as longas the Simulation Engine is generating real-time simulations instereoscopic 3D.

The above steps describe new and unique elements and process that makeupthe Hands-On Simulator with Multi-View capability. Multi-View providesthe end user with multiple and/or separate left- and right-eye views ofthe same simulation. Multi-View capability is a significant visual andinteractive improvement over the single eye view.

The present invention also allows the viewer to move around the threedimensional display and yet suffer no great distortion since the displaycan track the viewer eyepoint and re-display the images correspondingly,in contrast to the conventional prior art three dimensional imagedisplay where it would be projected and computed as seen from a singularviewing point, and thus any movement by the viewer away from theintended viewing point in space would cause gross distortion.

The display system can further comprise a computer capable ofre-calculate the projected image given the movement of the eyepointlocation. The horizontal perspective images can be very complex, tediousto create, or created in ways that are not natural for artists orcameras, and therefore require the use of a computer system for thetasks. To display a three-dimensional image of an object with complexsurfaces or to create animation sequences would demand a lot ofcomputational power and time, and therefore it is a task well suited tothe computer. Three dimensional capable electronics and computinghardware devices and real-time computer-generated three dimensionalcomputer graphics have advanced significantly recently with markedinnovations in visual, audio and tactile systems, and have producingexcellent hardware and software products to generate realism and morenatural computer-human interfaces. The horizontal perspective displaysystem of the present invention are not only in demand for entertainmentmedia such as televisions, movies, and video games but are also neededfrom various fields such as education (displaying three-dimensionalstructures), technological training (displaying three-dimensionalequipment). There is an increasing demand for three-dimensional imagedisplays, which can be viewed from various angles to enable observationof real objects using object-like images. The horizontal perspectivedisplay system is also capable of substitute a computer-generatedreality for the viewer observation. The systems may include audio,visual, motion and inputs from the user in order to create a completeexperience of three dimensional illusions.

The input for the horizontal perspective system can be two dimensionalimage, several images combined to form one single three dimensionalimage, or three dimensional model. The three dimensional image or modelconveys much more information than that a two dimensional image and bychanging viewing angle, the viewer will get the impression of seeing thesame object from different perspectives continuously.

The horizontal perspective display can further provide multiple views or“Multi-View” capability. Multi-View provides the viewer with multipleand/or separate left-and right-eye views of the same simulation.Multi-View capability is a significant visual and interactiveimprovement over the single eye view. In Multi-View mode, both the lefteye and right eye images are fused by the viewer's brain into a single,three-dimensional illusion. The problem of the discrepancy betweenaccommodation and convergence of eyes, inherent in stereoscopic images,leading to the viewer's eye fatigue with large discrepancy, can bereduced with the horizontal perspective display, especially for motionimages, since the position of the viewer's gaze point changes when thedisplay scene changes.

In Multi-View mode, the objective is to simulate the actions of the twoeyes to create the perception of depth, namely the left eye and theright eye sees slightly different images. Thus Multi-View devices thatcan be used in the present invention include methods with glasses suchas anaglyph method, special polarized glasses or shutter glasses,methods without using glasses such as a parallax stereogram, alenticular method, and mirror method (concave and convex lens).

In anaglyph method, a display image for the right eye and a displayimage for the left eye are respectively superimpose-displayed in twocolors, e.g., red and blue, and observation images for the right andleft eyes are separated using color filters, thus allowing a viewer torecognize a stereoscopic image. The images are displayed usinghorizontal perspective technique with the viewer looking down at anangle. As with one eye horizontal perspective method, the eyepoint ofthe projected images has to be coincide with the eyepoint of the viewer,and therefore the viewer input device is essential in allowing theviewer to observe the three dimensional horizontal perspective illusion.From the early days of the anaglyph method, there are much improvementssuch as the spectrum of the red/blue glasses and display to generatemuch more realism and comfort to the viewers.

In polarized glasses method, the left eye image and the right eye imageare separated by the use of mutually extinguishing polarizing filterssuch as orthogonally linear polarizer, circular polarizer, ellipticalpolarizer. The images are normally projected onto screens withpolarizing filters and the viewer is then provided with correspondingpolarized glasses. The left and right eye images appear on the screen atthe same time, but only the left eye polarized light is transmittedthrough the left eye lens of the eyeglasses and only the right eyepolarized light is transmitted through the right eye lens.

Another way for stereoscopic display is the image sequential system. Insuch a system, the images are displayed sequentially between left eyeand right eye images rather than superimposing them upon one another,and the viewer's lenses are synchronized with the screen display toallow the left eye to see only when the left image is displayed, and theright eye to see only when the right image is displayed. The shutteringof the glasses can be achieved by mechanical shuttering or with liquidcrystal electronic shuttering. In shuttering glass method, displayimages for the right and left eyes are alternately displayed on a CRT ina time sharing manner, and observation images for the right and lefteyes are separated using time sharing shutter glasses which areopened/closed in a time sharing manner in synchronism with the displayimages, thus allowing an observer to recognize a stereoscopic image.

Other way to display stereoscopic images is by optical method. In thismethod, display images for the right and left eyes, which are separatelydisplayed on a viewer using optical means such as prisms, mirror, lens,and the like, are superimpose-displayed as observation images in frontof an observer, thus allowing the observer to recognize a stereoscopicimage. Large convex or concave lenses can also be used where two imageprojectors, projecting left eye and right eye images, are providingfocus to the viewer's left and right eye respectively. A variation ofthe optical method is the lenticular method where the images form oncylindrical lens elements or two dimensional array of lens elements.

FIG. 27 is a horizontal perspective display focusing on how thecomputer-generated person's two eye views are projected onto theHorizontal Plane and then displayed on a stereoscopic 3D capable viewingdevice. FIG. 27 represents one complete display time period. During thisdisplay time period, the horizontal perspective display needs togenerate two different eye views, because in this example thestereoscopic 3D viewing device requires a separate left- and right-eyeview. There are existing stereoscopic 3D viewing devices that requiremore than a separate left- and right-eye view, and because the methoddescribed here can generate multiple views it works for these devices aswell.

The illustration in the upper left of FIG. 27 shows the Angled Camerapoint for the right eye after the first (right) eye-view to begenerated. Once the first (right) eye view is complete, the horizontalperspective display starts the process of rendering thecomputer-generated person's second eye (left-eye) view. The illustrationin the lower left of FIG. 27 shows the Angled Camera point for the lefteye after the completion of this time. But before the rendering processcan begin, the horizontal perspective display makes an adjustment to theAngled Camera point. This is illustrated in FIG. 27 by the left eye's xcoordinate being incremented by two inches. This difference between theright eye's x value and the left eye's x+2″ is what provides thetwo-inch separation between the eyes, which is required for stereoscopic3D viewing. The distances between people's eyes vary but in the aboveexample we are using the average of 2 inches. It is also possible forthe view to supply the horizontal perspective display with theirpersonal eye separation value. This would make the x value for the leftand right eyes highly accurate for a given viewer and thereby improvethe quality of their stereoscopic 3D view.

Once the horizontal perspective display has incremented the AngledCamera point's x coordinate by two inches, or by the personal eyeseparation value supplied by the viewer, the rendering continues bydisplaying the second (left-eye) view.

Depending on the stereoscopic 3D viewing device used, the horizontalperspective display continues to display the left- and right-eye images,as described above, until it needs to move to the next display timeperiod. An example of when this may occur is if the bear cub moves hispaw or any part of his body. Then a new and second Simulated Image wouldbe required to show the bear cub in its new position. This new SimulatedImage of the bear cub, in a slightly different location, gets renderedduring a new display time period. This process of generating multipleviews via the nonstop incrementing of display time continues as long asthe horizontal perspective display is generating real-time simulationsin stereoscopic 3D.

By rapidly display the horizontal perspective images, three dimensionalillusion of motion can be realized. Typically, 30 to 60 images persecond would be adequate for the eye to perceive motion. Forstereoscopy, the same display rate is needed for superimposed images,and twice that amount would be needed for time sequential method.

The display rate is the number of images per second that the displayuses to completely generate and display one image. This is similar to amovie projector where 24 times a second it displays an image. Therefore,1/24 of a second is required for one image to be displayed by theprojector. But the display time could be a variable, meaning thatdepending on the complexity of the view volumes it could take 1/12 or ½a second for the computer to complete just one display image. Since thedisplay was generating a separate left and right eye view of the sameimage, the total display time is twice the display time for one eyeimage.

FIG. 28 shows a horizontal plane as related to both central perspectiveand horizontal perspective.

The present invention hands-on simulator further includes technologiesemployed in computer “peripherals”. FIG. 29 shows examples of suchPeripherals with six degrees of freedom, meaning that their coordinatesystem enables them to interact at any given point in an (x, y, z)space. The simulator creates a “Peripheral Open-Access Volume,” for eachPeripheral the end-user requires, such as the Space Glove in FIG. 29.FIG. 30 is a high-level illustration of the Hands-On Simulation Tool,focusing on how a Peripheral's coordinate system is implemented withinthe Hands-On Simulation Tool.

The new Peripheral Open-Access Volume, which as an example in FIG. 30 islabeled “Space Glove,” is mapped one-to-one with the “Open-Access RealVolume” and “Open-Access Computer-generated Volume.” The key toachieving a precise one-to-one mapping is to calibrate the Peripheral'svolume with the Common Reference, which is the physical View surface,located at the viewing surface of the display device.

Some Peripherals provide a mechanism that enables the Hands-OnSimulation Tool to perform this calibration without any end-userinvolvement. But if calibrating the Peripheral requires externalintervention than the end-user will accomplish this through an“Open-Access Peripheral Calibration” procedure. This procedure providesthe end-user with a series of Simulations within the Hands-On Volume anda user-friendly interface that enables them to adjusting the location ofthe Peripheral's volume until it is in perfect synchronization with theView surface. When the calibration procedure is complete, the Hands-OnSimulation Tool saves the information in the end-user's personalprofile.

Once the Peripheral's volume is precisely calibrated to the Viewsurface, the next step in the process can be taken. The Hands-OnSimulation Tool will continuously track and map the Peripheral's volumeto the Open-Access Volumes. The Hands-On Simulation Tool modifies eachHands-On Image it generates based on the data in the Peripheral'svolume. The end result of this process is the end-user's ability to useany given Peripheral to interact with Simulations within the Hands-OnVolume generated in real-time by the Hands-On Simulation Tool.

With the peripherals linking to the simulator, the user can interactwith the display model. The Simulation Engine can get the inputs fromthe user through the peripherals, and manipulate the desired action.With the peripherals properly matched with the physical space and thedisplay space, the simulator can provide proper interaction and display.The invention Hands-On Simulator then can generate a totally new andunique computing experience in that it enables an end user to interactphysically and directly (Hands-On) with real-time computer-generated 3Dgraphics (Simulations), which appear in open space above the viewingsurface of a display device, i.e. in the end user's own physical space.The peripheral tracking can be done through camera triangulation orthrough infrared tracking devices.

The simulator can further include 3D audio devices for “SIMULATIONRECOGNITION & 3D AUDIO”. This results in a new invention in the form ofa Hands-On Simulation Tool with its Camera Model, Horizontal Multi-ViewDevice, Peripheral Devices, Frequency Receiving/Sending Devices, andHandheld Devices as described below.

Object Recognition is a technology that uses cameras and/or othersensors to locate simulations by a method called triangulation.Triangulation is a process employing trigonometry, sensors, andfrequencies to “receive” data from simulations in order to determinetheir precise location in space. It is for this reason thattriangulation is a mainstay of the cartography and surveying industrieswhere the sensors and frequencies they use include but are not limitedto cameras, lasers, radar, and microwave. 3D Audio also usestriangulation but in the opposite way 3D Audio “sends” or projects datain the form of sound to a specific location. But whether you're sendingor receiving data the location of the simulation in three-dimensionalspace is done by triangulation with frequency receiving/sending devices.By changing the amplitudes and phase angles of the sound waves reachingthe user's left and right ears, the device can effectively emulate theposition of the sound source. The sounds reaching the ears will need tobe isolated to avoid interference. The isolation can be accomplished bythe use of earphones or the like.

FIG. 31 shows an end-user looking at a Hands-On Image of a bear cub.Since the cub appears in open space above the viewing surface theend-user can reach in and manipulate the cub by hand or with a handheldtool. It is also possible for the end-user to view the cub fromdifferent angles, as they would in real life. This is accomplishedthough the use of triangulation where the three real-world camerascontinuously send images from their unique angle of view to the Hands-OnSimulation Tool. This camera data of the real world enables the Hands-OnSimulation Tool to locate, track, and map the end-user's body and otherreal-world simulations positioned within and around the computermonitor's viewing surface (FIG. 32).

FIG. 33 also shows the end-user viewing and interacting with the bearcub, but it includes 3D sounds emanating from the cub's mouth. Toaccomplish this level of audio quality requires physically combiningeach of the three cameras with a separate speaker, as shown in FIG. 32.The cameras' data enables the Hands-On Simulation Tool to usetriangulation in order to locate, track, and map the end-user's “leftand right ear”. And since the Hands-On Simulation Tool is generating thebear cub as a computer-generated Hands-On Image it knows the exactlocation of the cub's mouth. By knowing the exact location of theend-user's ears and the cub's mouth the Hands-On Simulation Tool usestriangulation to sends data, by modifying the spatial characteristics ofthe audio, making it appear that 3D sound is emanating from the cub'scomputer-generated mouth.

Create a new frequency receiving/sending device by combining a videocamera with an audio speaker, as previously shown in FIG. 31. Note thatother sensors and/or transducers may be used as well.

Take these new camera/speaker devices and attach or place them nearby aviewing device, such as a computer monitor as previously shown in FIG.32. This results in each camera/speaker device having a unique andseparate “real-world” (x, y, z) location, line-of-sight, and frequencyreceiving/sending volume. To understand these parameters think of usinga camcorder and looking through its view finder When you do this thecamera has a specific location in space, is pointed in a specificdirection, and all the visual frequency information you see or receivethrough the view finder is its “frequency receiving volume”.

Triangulation works by separating and positioning each camera/speakerdevice such that their individual frequency receiving/sending volumesoverlap and cover the exact same area of space. If you have three widelyspaced frequency receiving/sending volumes covering the exact same areaof space than any simulation within the space can accurately be located.The next step creates a new element in the Open-Access Camera Model forthis real-world space and in FIG. 33 it is labeled “real frequencyreceiving/sending volume”.

Now that this real frequency receiving/sending volume exists it must becalibrated to the Common Reference, which of course is the real ViewSurface. The next step is the automatic calibration of the realfrequency receiving/sending volume to the real View Surface. This is anautomated procedure that is continuously performed by the Hands-OnSimulation Tool in order to keep the camera/speaker devices correctlycalibrated even when they are accidentally bumped or moved by theend-user, which is likely to occur.

FIG. 34 is a simplified illustration of the complete Open-Access CameraModel and will assist in explaining each of the additional stepsrequired to accomplish the scenarios described in FIGS. 32 and 33 above.

The simulator then performs simulation recognition by continuouslylocating and tracking the end-user's “left and right eye” and their“line-of-sight”, continuously map the real-world left and right eyecoordinates into the Open-Access Camera Model precisely where they arein real space, and continuously adjust the computer-generated camerascoordinates to match the real-world eye coordinates that are beinglocated, tracked, and mapped. This enables the real-time generation ofSimulations within the Hands-On Volume based on the exact location ofthe end-user's left and right eye. Allowing the end-user to freely movetheir head and look around the Hands-On Image without distortion.

The simulator then perform simulation recognition by continuouslylocating and tracking the end-user's “left and right ear” and their“line-of-hearing”, continuously map the real-world left- and right-earcoordinates into the Open-Access Camera Model precisely where they arein real space, and continuously adjust the 3D Audio coordinates to matchthe real-world ear coordinates that are being located, tracked, andmapped. This enables the real-time generation of Open-Access soundsbased on the exact location of the end-user's left and right ears.Allowing the end-user to freely move their head and still hearOpen-Access sounds emanating from their correct location.

The simulator then perform simulation recognition by continuouslylocating and tracking the end-user's “left and right hand” and their“digits,” i.e. fingers and thumbs, continuously map the real-world leftand right hand coordinates into the Open-Access Camera Model preciselywhere they are in real space, and continuously adjust the Hands-On Imagecoordinates to match the real-world hand coordinates that are beinglocated, tracked, and mapped. This enables the real-time generation ofSimulations within the Hands-On Volume based on the exact location ofthe end-user's left and right hands allowing the end-user to freelyinteract with Simulations within the Hands-On Volume.

The simulator then perform simulation recognition by continuouslylocating and tracking “handheld tools”, continuously map thesereal-world handheld tool coordinates into the Open-Access Camera Modelprecisely where they are in real space, and continuously adjust theHands-On Image coordinates to match the real-world handheld toolcoordinates that are being located, tracked, and mapped. This enablesthe real-time generation of Simulations within the Hands-On Volume basedon the exact location of the handheld tools allowing the end-user tofreely interact with Simulations within the Hands-On Volume.

FIG. 35 is intended to assist in further explaining unique discoveriesregarding the new Open-Assess Camera Model and handheld tools. FIG. 35is a simulation of and end-user interacting with a Hands-On Image usinga handheld tool. The scenario being illustrated is the end-uservisualizing large amounts of financial data as a number of interrelatedOpen-Access 3D simulations. The end-user can probe and manipulated theOpen-Access simulations by using a handheld tool, which in FIG. 35 lookslike a pointing device.

A “computer-generated attachment” is mapped in the form of anOpen-Access computer-generated simulation onto the tip of a handheldtool, which in FIG. 35 appears to the end-user as a computer-generated“eraser”. The end-user can of course request that the Hands-OnSimulation Tool map any number of computer-generated attachments to agiven handheld tool. For example, there can be differentcomputer-generated attachments with unique visual and audiocharacteristics for cutting, pasting, welding, painting, smearing,pointing, grabbing, etc. And each of these computer-generatedattachments would act and sound like the real device they are simulatingwhen they are mapped to the tip of the end-user's handheld tool.

The present invention further discloses a Multi-Plane display comprisinga horizontal perspective display together with a non-horizontal centralperspective display. FIG. 36 illustrates an example of the presentinvention Multi-Plane display in which the Multi-Plane display is acomputer monitor that is approximately “L” shaped when open. Theend-user views the L-shaped computer monitor from its concave side andat approximately a 45° angle to the bottom of the “L,” as shown in FIG.36. From the end-user's point of view the entire L-shaped computermonitor appears as one single and seamless viewing surface. The bottom Lof the display, positioned horizontally, shows horizontal perspectiveimage, and the other branch of the L display shows central perspectiveimage. The edge is the two display segments is preferably smoothlyjoined and can also having a curvilinear projection to connect the twodisplays of horizontal perspective and central perspective.

The Multi-Plane display can be made with one or more physical viewingsurfaces. For example, the vertical leg of the “L” can be one physicalviewing surface, such as flat panel display, and the horizontal leg ofthe “L” can be a separate flat panel display. The edge of the twodisplay segments can be a non-display segment and therefore the twoviewing surface are not continuous. Each leg of a Multi-Plane display iscalled a viewing plane and as you can see in the upper left of FIG. 36there is a vertical viewing plane and a horizontal viewing plane where acentral perspective image is generated on the vertical plane and ahorizontal perspective image is generated on the horizontal plane, andthen blend the two images where the planes meet, as illustrated in thelower right of FIG. 36.

FIG. 36 also illustrates that a Multi-Plane display is capable ofgenerating multiple views. Meaning that it can display single-viewimages, i.e. a one-eye perspective like the simulation in the upperleft, and/or multi-view images, i.e. separate right and left eye viewslike the simulation in the lower right. And when the L-shaped computermonitor is not being used by the end-user it can be closed and look likethe simulation in the lower left.

FIG. 37 is a simplified illustration of the present inventionMulti-Plane display. In the upper right of FIG. 37 is an example of asingle-view image of a bear cub that is displayed on an L-shapedcomputer monitor. Normally a single-view or one eye image would begenerated with only one camera point, but as you can see there are atleast two camera points for the Multi-Plabe display even though this isa single-view example. This is because each viewing plane of aMulti-Plane device requires its own rendering perspective. One camerapoint is for the horizontal perspective image, which is displayed on thehorizontal surface, and the other camera point is for the centralperspective image, which is displayed on the vertical surface.

To generate both the horizontal perspective and central perspectiveimages requires the creation of two camera eyepoints (which can be thesame or different) as shown in FIG. 37 for two different and separatecamera points labeled OSI and CPI. The vertical viewing plane of theL-shaped monitor, as shown at the bottom of FIG. 37, is the displaysurface for the central perspective images, and thus there is a need todefine another common reference plane for this surface. As discussedabove, the common reference plane is the plane where the images aredisplay, and the computer need to keep track of this plane for thesynchronization of the locations of the displayed images and the realphysical locations. With the L-shaped Multi-Plane device and the twodisplay surfaces, the Simulation can to generate the three dimansionalimages, a horizontal perspective image using (OSI) camera eyepoint, anda central perspective image using (CPI) camera eyepoint.

The multi-plane display system can further include a curvilinearconnection display section to blend the horizontal perspective and thecentral perspective images together at the location of the seam in the“L,” as shown at the bottom of FIG. 37. The multi-plane display systemcan continuously update and display what appears to be a single L-shapedimage on the L-shaped Multi-Plane device.

Furthermore, the multi-plane display system can comprise multipledisplay surfaces together with multiple curvilinear blending sections asshown in FIG. 38. The multiple display surfaces can be a flat wall,multiple adjacent flat walls, a dome, and a curved wraparound panel.

The present invention multi-plane display system thus can simultaneouslyprojecting a plurality of three dimensional images onto multiple displaysurfaces, one of which is a horizontal perspective image. Further, itcan be a stereoscopic multiple display system allowing viewers to usetheir stereoscopic vision for three dimensional image presentation.

Since the multi-plane display system comprises at least two displaysurfaces, various requirements need to be addressed to ensure highfidelity in the three dimensional image projection. The displayrequirements are typically geometric accuracy, to ensure that objectsand features of the image to be correctly positioned, edge matchaccuracy, to ensure continuity between display surfaces, no blendingvariation, to ensure no variation in luminance in the blending sectionof various display surfaces, and field of view, to ensure a continuousimage from the eyepoint of the viewer.

Since the blending section of the multi-plane display system ispreferably a curve surface, some distortion correction could be appliedin order for the image projected onto the blending section surface toappear correct to the viewer. There are various solutions for providingdistortion correction to a display system such as using a test patternimage, designing the image projection system for the specific curvedblending display section, using special video hardware, utilizing apiecewise-linear approximation for the curved blending section. Stillanother distortion correction solution for the curve surface projectionis to automatically computes image distortion correction for any givenposition of the viewer eyepoint and the projector.

Since the multi-plane display system comprises more than one displaysurface, care should be taken to minimize the seams and gaps between theedges of the respective displays. To avoid seams or gaps problem, therecould be at least two image generators generating adjacent overlappedportions of an image. The overlapped image is calculated by an imageprocessor to ensure that the projected pixels in the overlapped areasare adjusted to form the proper displayed images. Other solutions are tocontrol the degree of intensity reduction in the overlapping to create asmooth transition from the image of one display surface to the next.

The three dimensional simulator would not be complete without a threedimensional audio or binaural simulation. Binaural simulation offersrealism to the three dimensional simulation together with 3Dvisualization.

Similar to vision, hearing using one ear is called monoaural and hearingusing two ears is called binaural. Hearing can provide the direction ofthe sound sources but with poorer resolution than vision, the identityand content of a sound source such as speech or music, and the nature ofthe environment via echoes, reverberation such as a normal room or anopen field.

The head and ears, and sometime the shoulder, function as an antennasystem to provide information about the location, distance andenvironment of the sound sources. The brain can interprete properly thevarious kinds of sound arriving at the head such as direct sounds,diffractive sounds around the head and by interaction with the outerears and shoulder, different sound amplitudes and different arrival timeof the sounds. These acoustic modifications are called ‘sound cues’ andserve to provide us the directional acoustis information of the sounds.

Basically, the sound cues are related to timing, volume, frequency andreflection. In timing cues, the ears recognize the time the soundarrives and assume that the sound comes from the closest source.Further, with two ears separated about 8 inches apart, the delay of thesound reaching one ear with respect to the other ear can give a cueabout the location of the sound source. The timing cue is stronger thanthe level cue in the sense that the listener locates the sound based onthe first wave that reaches the ear, regardless of the loudness of anylater arriving waves. In volume (or level) cues, the ears recognize thevolume (or loudness) of the sound and assume that the sound coming fromthe loudest direction. With the binaural (two ears) effect, theamplitude difference between the ears is a strong cue for thelocalization of the sound source. In frequency (or equalization) cues,the ears recognize the frequency balance of the sound as it arrives ineach ear since frontal sounds are directed into the eardrums, while rearsounds bounce off the external ear and thus having a high frequency rolloff. In reflection cue, the sound bounces off various surfaces and areeither dispersed or absorbed in varying degrees before reaching the earsmultiple times. This reflections off the walls of the room and theforeknowledge of the difference between the way various floor coveringssound also contribute to localization. In addition, the body, especiallythe head, can move relative to the sound source to help in locate thesound.

The above various sound cues are scientifically classified into threetypes of spatial hearing cues: interaural time differences (ITDs),interaural level differences (ILDs), and head-related transfer functions(HRTFs). ITDs relate to the time for a sound to reach the ears and thetime difference for reaching both ears. ILDs refer to the amplitude inthe frequency spectrum of sound reaching the ears and also the amplitudedifferences of the sound frequencies as heard in both ears. HRTFs canprovide the perception of distance by the changes in the timbre anddistance dependencies, the time delay and directions of direct sound andreflections in echoic environments.

The HRTFs are a collection of spatial cues for a particular listener,including ITDs, ILDs and the reflections, diffractions and dampingcaused by the listener's body, head, outer ears and shoulder. Theexternal ear, or pinna, has a significant contribution to the HRTFs.Higher frequency sounds are filtered by the pinna to provide the brain away as to perceive the lateral position, or azimuth, and elevation ofthe sound source since the response of the pinna filter is highlydependent on the overall direction of the sound source. The head canaccount for a reduced amplitude of various frequencies of the soundssince the sound has to go through or around the head in order to reachthe ear. The overall effects of head shadowing contribute to theperception of linear distance and direction of a sound source. Further,sound frequencies in the range of 1-3 kHz are reflected from theshoulder to produce echoes representing a time delay dependent on theelevation of the sound source. The reflections from surfaces in theworld and the reverberation also seem to affect the localizationjudgement of sound distance and direction.

In addition to these cues, the movement of the head to help in locatethe location of a sound source is a key factor, together with the visionto confirm the sound direction. For a 3D immersion, all mechanisms tolocalize the sounds are always in play and should normally agree. Ifnot, there would be some discomfort and confusion.

Although we can hear with one ear, hearing with two ears is clearlybetter. Many of the sound cues are related to the binaural perceptiondepending on both the relative loudness of sound and the relative timeof arrival of sound at each ear. And thus the binaural performance isclear superior for the localization of single or multiple sound sourcesand for the formation of the room environment, for the separation ofsignals coming from multiple incoherent and coherent sound sources; andthe enhancement of a chosen signal in a reverberant environment.

Mathematically speaking, HRTF is the frequency response of the soundwaves as received by the ears. By measuring the HRTF of a particularlistener, and by synthesised electronically using digital signalprocessing, the sounds can be delivered to a listener's ears viaheadphones or loudspeakers to create a virtual sound image in threedimensions.

The sound transformation to the ear canal, i.e. HRTF frequency response,can be measured accurately by using small microphones in the ear canals.The measured signal is then processed by a computer to derive the HRTFfrequency responses for the left and right ears corresponding to thesound source location.

Thus a 3D audio system works by using the measured HRTFs as the audiofilters or equalizers. When a sound signal is processed by the HRTFsfilters, the sound localization cues are reproduced, and the listenershould perceive the sound at the location specified by the HRTFs. Thismethod of binaural synthesis works extremely well when the listener'sown HRTFs are used to synthesize the localization cues. However,measuring HRTFs is a complicated procedure, so 3D audio systemstypically use a single set of HRTFs previously measured from aparticular human or manikin subject. Thus the HRTF sometimes needs to bechanged to accurately respond to a perticular listener. The tuning of aHRTF function can be accomplished by providing various sound sourcelocations and environments and asking the listener to identify.

A 3D audio system should provide the ability for the listener to definea three-dimensional space, to position multiple sound sources and thatlistener in that 3D space, and to do it all in real-time, orinteractively. Beside 3D audio system, other technologies such stereoextension and surround sound could offer some aspects of 3D positioningor interactivity.

Extended stereo processes an existing stereo (two channel) soundtrack toadd spaciousness and to make it appear to originate from outside theleft/right speaker locations through fairly straight-forward methods.Some of the characteristics of the extended stereo technology includethe size of the listening area (called sweet spot), the amount ofspreading of stereo images, the amount of tonal changes, the amount oflost stereo panning information, and the ability to achieve effect onheadphones as well as speakers.

The surround sound create a larger-than-stereo sound stage with asurround sound 5-speaker setup. Additionally, virtual surround soundsystems use 3D audio technology to create the illusion of five speakersemanating from a regular set of stereo speakers, therefore enabling asurround sound listening experience without the need for a five speakersetup. The characteristics of the surround sound technology include thepresentation accuracy, the clarity of spatial imaging, and the size ofthe listening area

For better 3D audio system, audio technology needs to create a life-likelistening experience by replicating the 3D audio cues that the ears hearin the real world for allowing non-interactive and interactive listeningand positioning of sounds anywhere in the three-dimensional spacesurrounding a listener.

The head tracker function is also very important to provide perceptualroom constancy to the listener. In other words, when the listener movetheir heads around, the signals would change so that the perceivedauditory world maintain its spatial position. To this end, thesimulation system needs to know the head position in order to be able tocontrol the binaural impulse responses adequately. Head position sensorshave therefore to be provided. The impression of being immersed is ofparticular relevance for applications in the context of virtual reality.

A replica of a sound field can be produced by putting an infinite numberof microphones everywhere. After being stored on a recorder with aninfinite number of channels, this recording can then be played backthrough an infinite number of point-source loudspeakers, each placedexactly as its corresponding microphone was placed. As the number ofmicrophones and speakers is reduced, the quality of the sound fieldbeing simulated suffers. By the time we are down to two channels, heightcues have certainly been lost and instead of a stage that is audiblefrom anywhere in the room we find that sources on the stage are now onlylocalizable if we listen along a line equidistant from the last tworemaining speakers and face them.

However, only two channels should be adequate, since if we deliver theexact sound required to simulate a live performance at the entrance toeach ear canal, then since we only have two ear canals, we should onlyneed to generate two such sound fields. In other words, since we canhear three-dimensionally in the real world using just two ears, it mustbe possible to achieve the same effect from just two speakers or a setof headphones.

Headphone reproduction is thus differed from loudspeaker reproductionsince headphone microphones should be spaced about seven inches apartfor a normal ear separation, and loudspeaker microphones separationshould be about seven feet apart. Further loudspeakers suffer fromcrosstalk and therefore some signal conditioning such as crosstalkcancellation will be needed for 3D loudspeaker setup.

Loudspeaker 3D audio systems are extremely effective in desktopcomputing environments. This is because there is usually only a singlelistener (the computer user) who is almost always centered between thespeakers and facing forward towards the monitor. Thus, the primary usergets the full 3D effect because the crosstalk is properly cancelled. Intypical 3D audio applications, like video gaming, friends may gatheraround to watch. In this case, the best 3D audio effects are heard byothers when they are also centered with respect to the loudspeakers.Off-center listeners may not get the full effect, but they still hear ahigh quality stereo program with some spatial enhancements.

To achieve 3D audio, the speakers are typically arranged surrounding thelistener in about the same horizontal plane, but could be arranged tocompletely surround the listener, from the ceiling to the floor to thesurrounding walls. Optionally, the speakers can also be put on theceiling, on the floor, arranged in an overhead dome configuration, orarranged in a vertical wall configuration. Further, beam transmittedspeakers can be used instead of headphone. Beam transmitted speakeroffers the freedom of movement for the listener and without thecrosstalk between speakers since beam transmitted speaker provide atight beam of sound.

Generally, a minimum of four loudspeakers are required to achieve aconvincing 3-D audio experience, while some researchers are using twentyor more speakers in an anechoic chamber to recreate acousticenvironments with much greater precision.

The main advantages of multi-speaker playback are:

-   -   There is no dependence on the individual subject's HRTF, since        the sound field is created without any reference to individual        listeners.    -   The subject is free to turn their head, and even move about        within a limited range.    -   In some cases, more than one subject can listen to the system        simultaneously.

Many crosstalk cancellers are based on a highly simplified model ofcrosstalk, for example modeling crosstalk as a simple delay andattenuation process, or a delay and a lowpass filter. Other crosstalkcancellers have been based on a spherical head model. As with binauralsynthesis, crosstalk cancellation performance is ultimately limited bythe variation in the size and shape of human heads. 3D audio simulationcan be accomplished by the following steps:

-   -   Input the characteristics of the acoustic space.    -   Determine the sequence of sound arrivals that occur at the        listening position. Each sound arrival will have the following        characteristics: (a) time of arrival, based on the distance        travelled by the echo-path, (b) direction of arrival, (c)        attenuation (as a function of frequency) of the sound due to the        absorption properties of the surfaces encountered by the        echo-path.    -   Compute the impulse response of the acoustic space incorporating        the multiple sound arrivals.    -   The results from the FIR filter are played back to a listener.        In the case where the impulse responses were computed using a        dummy head response, the results are played over headphones to        the listener. In this case, the equalisation required for the        particular headphones is also applied.

The simulation of an acoustic environment involves one or more of thefollowing functions:

-   -   Processing an audio source input and presenting it to the        subject through a number of loudspeakers (or headphones) with        the intention of making the sound source appear to be located at        a particular position in space.    -   Processing multiple input audio sources in such a way that each        source is independently located in space around the subject.    -   Enhanced processing to simulate some aspects of the room        acoustics, so that the user can acoustically sense the size of        the room and the nature of the floor and wall coverings.    -   The capability for the subject to move (perhaps within a limited        range) and turn his/her head so as to focus attention on some        aspects of the sound source characteristics or room acoustics.

Binaural simulation is generally carried out using the sound sourcematerial free from any unwanted echoes or noise. The sound sourcematerial can then be replayed to a subject, using the appropriate HRTFfilters, to create the illusion that the source audio is originatingfrom a particular direction. The HRTF filtering is achieved by simplyconvolving the audio signal with the pair of HRTF responses (one HRTFfilter for each channel of the headphone). The eyes and ears oftenperceive an event at the same time. Seeing a door close, and hearing ashutting sound, are interpreted as one event if they happensynchronously. If we see a door shut without a sound, or we see a doorshut in front of us, and hear a shutting sound to the left, we getalarmed and confused. In another scenario, we might hear a voice infront of us, and see a hallway with a corner; the combination of audioand visual cues allows us to figure out that a person might be standingaround the corner. Together, synchronized 3D audio and 3D visual cuesprovide a very strong immersion experience. Both 3D audio and 3Dgraphics systems can be greatly enhanced by such synchronization.

Improved playback through headphones can be achieved through the use ofhead tracking. This technique makes use of continuous measurements ofthe orientation of a subject's head, and adapts the audio signals beingfed to the headphones appropriately. Binaural signal should allow asubject to easily discriminate between left and right sound sourcelocations easily, but the ability to discriminate between front andback, and high and low sound sources is generally only possible if headmovement is permitted. Whilst multiple speaker playback methods solvethis problem to a large degree, there are still many applications whereheadphone playback is preferable, and head tracking can then be used asa valuable tool for improving the quality of the 3-D playback.

The simplest form of head tracking binaural system is one which simplysimulates anechoic HRTFs, and changes the HRTF functions rapidly inresponse to the subjects head movements. This HRTF switching can beachieved through a lookup table, with interpolation used to resolveangles that are not represented in the HRTF table.

Simulation of room acoustics over headphones with head tracking becomesmore difficult because the direction of arrival of the early reflectionsis also important in making the result sound realistic. Many researchersbelieve that the echoes in the reverberant tail of the room response aregenerally so diffuse that there is no requirement for this part of theroom response to be tracked with the subject's head movements.

An important feature of any head tracking playback system is the delayfrom the subject head movement to the change in the audio response atthe headphones. If this delay is excessive, the subject can experience aform of virtual motion sickness and general disorientation.

Audio cues change dramatically when a listener tilts or rotates his orher head. For example, quickly turning the head 90 degrees to look tothe side is the equivalent of a sound traveling from the listener's sideto the front in a split second. We often use head motion to track soundsor to search for them. The ears alert the brain about an event outsideof the area that the eyes are currently focused on, and we automaticallyturn to redirect our attention. Additionally, we use head motion toresolve ambiguities: a faint, low sound could be either in front or backof us, so we quickly and sub-consciously turn our head a small fractionto the left, and we know if the sound is now off to the right, it is inthe front, otherwise it is in the back. One of the reasons whyinteractive audio is more realistic than pre-recorded audio(soundtracks) is the fact that the listeners head motion can be properlysimulated in an interactive system (using inputs from a joystick, mouse,or head-tracking system).

The HRTF function are performed using digital signal processing (DSP)hardware for real time performance. Typical feature of DSP are that thedirect sound must be processed to give the correct amplitude andperceived direction, the early echoes must arrive at the listener withappropriate time, amplitude and frequency response to give theperception of the size of the spaces (as well as the acoustic nature ofthe room surfaces), and the late reverberation must be natural andcorrectly distributed in 3-D around the listener. The relative amplitudeof the direct sound compared to the remainder of the room response helpsto provide the sensation of distance.

Thus 3D audio simulation can provides a binaural gain so that the exactsame audio content is more audible and intelligible in the binauralcase, because the brain can localize and therefore “single out” thebinaural signal, while the non-binaural signal gets washed into thenoise. Further the listener would still be able to tune into andunderstand individual conversations, because they are still spatiallyseparated, and “amplified by” binaural gain, an effect called thecocktail party effct. Binaural simulation also can provide fasterreaction time because such a signal mirrors the ones received in thereal world. In addition, binaural signals can convey positionalinformation: a binaural radar warning sound can warn a user about aspecific object that is approaching (with a sound that is unique to thatobject), and naturally indicate where that object is coming from. Alsolistening to binaural simulation can beless fatigue since we are used tohearing sounds that originate outside of their heads, as is the casewith binaural signals. Mono or stereo signals appear to come from insidea listener's head when using headphones, and produce more strain than anatural sounding, binaural signal. An lastly, 3D binaural simulation canprovide an increased perception and immersion in higher quality 3Denvironment when visuals are shown in synch with binaural sound.

1. A method for 3-D horizontal perspective simulation by horizontalperspective projection, the horizontal perspective projection comprisingdisplaying horizontal perspective images according to a predeterminedprojection eyepoint, the method comprising the steps of: displaying a3-D image onto an open space of a first display surface using horizontalperspective; presenting 3-D sound to a predetermined projection earpointcorresponding to the 3-D image; and manipulating the display image onthe first display surface by touching the 3-D image with a peripheraldevice.
 2. A method for 3-D horizontal perspective simulation byhorizontal perspective projection, the horizontal perspective projectioncomprising displaying horizontal perspective images according to apredetermined projection eyepoint, the method comprising the steps of:displaying a 3-D image onto an open space of a first display surfaceusing horizontal perspective; display a second image onto a seconddisplay; presenting 3-D sound to a predetermined projection earpointcorresponding to the 3-D image; and manipulating the display image onthe first display surface by touching the 3-D image with a peripheraldevice.
 3. A method as in claim 2 wherein presenting 3-D sound comprisesoutputting two channel sound through a HRTF (head related transferfunction) filter.
 4. A method as in claim 2 wherein presenting 3-D soundcomprises outputting sound through a 3-D loudspeaker audio system or a3-D headphone audio system.
 5. A method as in claim 2 further comprisingthe step of taking an input from the second display and providing outputto the first horizontal perspective display.
 6. A method as in claim 2further comprising a step of tracking the physical peripheral device tothe 3-D image.
 7. A method as in claim 6 wherein tracking the peripheraldevice comprises tracking a tip of the peripheral device.
 8. A method asin claim 6 wherein the peripheral device tracking comprises inputtingthe position of the peripheral device to the processing unit.
 9. Amethod as in claim 6 wherein the peripheral device tracking comprises astep of triangulation or infrared tracking.
 10. A method as in claim 2further comprising a step of display a third image onto a thirdcurvilinear display, the curvilinear display blending the first displayand the second display.
 11. A method as in claim 2 wherein thehorizontal perspective display is a stereoscopic horizontal perspectivedisplay using horizontal perspective to display a stereoscopic 3-Dimage.
 12. A method as in claim 2 further comprising a step of automaticor manual eyepoint tracking for the horizontal perspective display. 13.A method as in claim 12 further wherein the eyepoint tracking furtheracts as an earpoint tracking.
 14. A method as in claim 2 furthercomprising a step of automatic or manual earpoint tracking for the 3-Dsound projection.
 15. A method as in claim 2 further comprising a stepof zooming, rotating or moving the 3-D image.
 16. A method as in claim 2wherein manipulating the display image by the peripheral devicecomprises tracking a tip of the peripheral device.
 17. A method as inclaim 16 wherein the manipulation comprises the action of modifying thedisplay image or the action of generating a different image.
 18. A 3-Dsimulation method using a 3-D horizontal perspective simulator system,the 3-D horizontal perspective simulator system comprising a processingunit; a first horizontal perspective display using horizontalperspective to display a 3-D image onto an open space; a second displayshowing information related to the 3-D image; a 3-D audio simulationsystem providing 3-D sound to a predetermined projection earpoint; aperipheral device to manipulate the display image by touching the 3-Dimage; and a peripheral device tracking unit for mapping the peripheraldevice to the 3-D image; the method comprising calibrating theperipheral device; displaying a first 3-D image onto an open space ofthe first display surface using horizontal perspective; displaying asecond image onto the second display; presenting 3-D sound correspondingto the 3-D image; tracking the peripheral device; and manipulating thedisplay image by touching the 3-D image with the peripheral device. 19.A method as in claim 18 wherein the 3-D audio simulation systemcomprises two sound channels and a HRTF (head related transfer function)filter.
 20. A method as in claim 18 further comprising a step of displaya third image onto a third curvilinear display, the curvilinear displayblending the first display and the second display.